Machine learning inventory management

ABSTRACT

Systems and methods for machine learning inventory management. The methods comprise: capturing images by a plurality of image capture devices of different types (e.g., visual camera, 3D camera and/or thermal camera); reading item identification codes for items represented in the images; and using at least a first portion of the images and known physical appearances of a plurality of items by a machine learning algorithm to learn relationships between the items represented in the images and the item identification codes. At least a second portion of the images are used to learn various types of information that is useful for inventory management (e.g., changes in inventory amounts for display equipment, changes in equipment cleanliness, changes in inventory packaging, item misplacements, etc.).

FIELD

This document relates generally to inventory management systems. Moreparticularly, this document relates to systems and methods for providingmachine learning inventory management.

BACKGROUND

There are many aspects to maintaining good inventory control in a store.Product placement, attractively arranged products, proper grouping ofproducts and sufficient products on the shelves all greatly contributeto increased sales and customer satisfaction. Products on shelves arenot static and are constantly changing as customers examine, misplaceand purchase items. It takes a significant amount of employee effort tomaintain the product displays at their optimum level. Sometimesmisplaced items are only discovered during inventory cycle counts.

The current methods are typically either employees or brand managerswalk the aisles or sometime RFID enabled systems can notify employees oflow values based upon RFID scanned sales. These methods are cumbersome,expensive or only provide partial coverage for the inventory.

SUMMARY

The present disclosure concerns implementing systems and methods formachine learning inventory management. The methods comprise: capturingimages by a plurality of image capture devices of different types (e.g.,a visual camera, a 3D camera, and/or a thermal camera); reading itemidentification codes for items represented in the images; and using atleast a first portion of the images and known physical appearances of aplurality of items by a machine learning algorithm to learnrelationships between the items represented in the images and the itemidentification codes.

In some scenarios, Point of Sale (“POS”) transaction information is usedto learn patterns of changes in amounts of inventory for given itemsover time. At least a second portion of the images which were capturedat the POS are used to identify items represented therein based on thelearned relationships between the items represented in the first portionof images and the item identification codes. The images, the knownphysical appearances, the item identification codes, and the POStransaction information may also be used to detect theft and/or learnpatterns of theft over time. The images are further be analyzed to learnchanges in item packaging. The changes in image packaging can further belearned based on at least one of the item identification codes, POStransaction information, and stored information for display equipmentinventory.

In those or other scenarios, at least a second portion of the images areused to learn at least one of: a change in a type of item disposed on apiece of equipment; a change in an amount of an item disposed on a pieceof equipment; conditions of equipment or areas adjacent to theequipment; individuals' interactions with equipment and inventorydisposed on the equipment; when equipment has been moved to a givenlocation; and/or when an item has been misplaced. If the system learnsthat the equipment has been moved to a given location, then learnedinformation therefor may be reset.

DESCRIPTION OF THE DRAWINGS

The present solution will be described with reference to the followingdrawing figures, in which like numerals represent like items throughoutthe figure.

FIG. 1 is a schematic illustration of an illustrative inventory system.

FIG. 2 is an illustration of an illustrative display equipment and imagecapture device arrangement.

FIG. 3 is an illustration of an illustrative POS device and imagecapture device arrangement.

FIG. 4 is an illustration of an illustrative passageway and imagecapture device arrangement.

FIG. 5 is a block diagram of an illustrative electronic device.

FIGS. 6A-6B (collectively referred to as “FIG. 6”) provide a flowdiagram of an illustrative method for video machine learning inventorymanagement.

FIG. 7 is an illustration of an illustrative method for machine learninginventory management.

DETAILED DESCRIPTION

It will be readily understood that the components of the presentsolution as generally described herein and illustrated in the appendedfigures could be arranged and designed in a wide variety of differentconfigurations. Thus, the following more detailed description of thepresent solution, as represented in the figures, is not intended tolimit the scope of the present disclosure, but is merely representativeof various implementations. While the various aspects of the presentsolution are presented in drawings, the drawings are not necessarilydrawn to scale unless specifically indicated.

The present solution may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the present solution is, therefore,indicated by the appended claims rather than by this detaileddescription. All changes which come within the meaning and range ofequivalency of the claims are to be embraced within their scope.

Reference throughout this specification to features, advantages, orsimilar language does not imply that all of the features and advantagesthat may be realized with the present solution should be or are in anysingle embodiment of the present solution. Rather, language referring tothe features and advantages is understood to mean that a specificfeature, advantage, or characteristic described in connection with anembodiment is included in at least one embodiment of the presentsolution. Thus, discussions of the features and advantages, and similarlanguage, throughout the specification may, but do not necessarily,refer to the same embodiment.

Furthermore, the described features, advantages and characteristics ofthe present solution may be combined in any suitable manner in one ormore embodiments. One skilled in the relevant art will recognize, inlight of the description herein, that the present solution can bepracticed without one or more of the specific features or advantages ofa particular embodiment. In other instances, additional features andadvantages may be recognized in certain embodiments that may not bepresent in all embodiments of the present solution.

Reference throughout this specification to “one embodiment”, “anembodiment”, or similar language means that a particular feature,structure, or characteristic described in connection with the indicatedembodiment is included in at least one embodiment of the presentsolution. Thus, the phrases “in one embodiment”, “in an embodiment”, andsimilar language throughout this specification may, but do notnecessarily, all refer to the same embodiment.

As used in this document, the singular form “a”, “an”, and “the” includeplural references unless the context clearly dictates otherwise. Unlessdefined otherwise, all technical and scientific terms used herein havethe same meanings as commonly understood by one of ordinary skill in theart. As used in this document, the term “comprising” means “including,but not limited to”.

There is a need for a more generic and automated method of monitoringdisplay equipment stocking. Accordingly, the present solution providessystems and methods for machine learning inventory management. Themachine learning is achieved using captured images and/or videos.Notably, videos comprise a sequence of images. As such, the presentsolution is described herein in terms of images. This description isintended to encompass video applications as well.

The methods involve continuously monitoring display equipment (e.g.,shelves) and/or POS devices using a network of image capturing devices(e.g., 3D, visual and IR cameras). With machine learning, the network isable to continuously monitor a facility (e.g., a retail store) and learn(a) how each piece of display equipment (e.g., shelve) should appear atdifferent times of day (e.g., the opening and/or closing of thefacility), (b) how each piece of display equipment should appear asinventory is removed therefrom, and (c) when restocking of the piece ofdisplay equipment should be performed. The network is also able todetect and identify misplaced items by color, pattern, size, heatsignatures, and/or location. The network is further able to determinewhen a piece of display equipment needs to be cleaned and/or determinewhen items disposed on the piece of display equipment need to berearranged/reorganized to put the piece of display equipment back in itsoptimal state.

An inventorying system implementing the present solution is able tooperate without having knowledge of the inventory items other than theirphysical appearances and/or initial locations within a facility. Byadding image capture devices at POS stations and/or other item checkingstations disposed on the sales floor, the captured images can becorrelated to item identification information obtained from productlabels (e.g., barcodes) and/or product markers (e.g., Radio FrequencyIdentification (“RFID”) tags). This correlation facilitates the machinelearning feature of the present solution to know and report productidentification codes (e.g., Universal Product Codes (“UPCs”) as well.Knowledge of the product identification codes helps with restockingsince individuals (e.g., employees) are not required to go to thelocation to determine what product(s) is(are) missing from the displayequipment (e.g., shelve).

The inventorying system is able to determine normal and abnormal displayequipment (e.g., shelve) conditions through machine learning, as well asalert/notify individuals (e.g., employees) of display equipment states.Alert/notification verifications and/or challenges by individuals canalso be used as machine learning feedback so that the inventoryingsystem continues to refine when alert/notification issuance is proper.

Referring now to FIG. 1, there is provided an illustration of anillustrative inventory system 100. Inventory system 100 is entirely orat least partially disposed within a facility 102. The facility 102 caninclude, but is not limited to, a manufacturer's facility, adistribution center facility, a retail store facility or other facilitywithin a supply chain.

As shown in FIG. 1, at least one item 118 resides within the facility102. The item 118 is disposed on display equipment 122. The displayequipment includes, but is not limited to, shelves 106 ₁-106 _(N),display cabinets, and/or exhibit cases. The item 118 has a marker (e.g.,an EAS security tag, or an RFID security tag) 120 and/or label 108(e.g., barcode) coupled thereto. This coupling is achieved via anadhesive (e.g., glue), a mechanical coupler (e.g., straps, clamps,snaps, etc.), a weld, a chemical bond or other means.

Markers and labels are well known in the art, and therefore will not bedescribed in detail here. Generally, the marker 120 and/or label 108is(are) configured to visually and/or auditorily present item levelinformation to people located in proximity thereto (e.g., customers).The information can include, but is not limited to, an item description,item nutritional information, a promotional message, an item regularprice, an item sale price, a currency symbol, and/or a source of theitem. Any known or to be known type of marker and/or label can be usedherein without limitation.

A plurality of image capture devices 104 are also provided withinfacility 102. The image capture devices include, but are not limited to,visual cameras, 3D cameras, and/or IR cameras. Each of the listed typesof cameras are well known in the art, and therefore are not described indetail herein. Any known or to be known visual, 3D and/or IR camera canbe used herein without limitation. For example, the visual cameraincludes a digital video camera configured to produce a 2D image ofobject(s). The 3D camera includes a 3D video camera configured toproduce 3D images indicating the distance between object(s) and areference point. The IR camera includes a thermal imaging cameraconfigured to produce thermal images. The images produced by these threetypes of cameras are collectively used herein for machine learninginventory management, as discussed herein. The image capture devices 104can be fixed or mobile. In the mobile scenarios, the image capturedevice(s) comprise a robot (e.g., an Unmanned Ground Vehicle (“UGV”)with a camera attached thereto) or a drone (e.g., an Unmanned AerialVehicle (“UAV”) with a camera attached thereto).

The image capture devices 104 are strategically placed throughout thefacility. For example, as shown in FIG. 2, image capture devices 104 ₁,104 ₂, 104 ₃, 104 ₄, 104 ₅, 104 ₆ are respectively placed in proximityto display equipment 122 ₁, 122 ₂. In the case that the image captureddevices are fixed devices, each image capture device 104 ₁-104 ₆ ismounted to the ceiling above the display equipment 122 ₁, 122 ₂ ormounted on the display equipment 122 ₁, 122 ₂ such that at least aportion of an opposing piece of display equipment is within its Field OfView (“FOV”) 204. As shown in FIG. 3, at least one image capture device104 ₇ is disposed in proximity to a transaction location 126 ₁ such thatit can monitor activity in a geographic area where a POS device 126 ₁resides. As shown in FIG. 4, an image capture device 104 ₈ is disposedin proximity to a passageway 402 of the facility 102. The presentsolution is not limited to the particular image capture deviceplacements shown in FIGS. 2-4. Other capture devicearrangements/placements can be used herein without limitation inaccordance with a particular application. For example, one or more imagecapture devices can additionally or alternatively be disposed inproximity to an item checking device disposed within the facility (e.g.,on the sales floor of the facility). Image capture devices 104 ₁-104 ₈are collectively referred to herein as image capture devices 104.

The image capture devices 104 and POS devices 126 are communicativelycoupled to a computing device 112 via a network 110. The communicationscan be achieved using wired or wireless communications links. Thenetwork includes, but is not limited to, an Intranet and/or theInternet. The computing device 112 includes, but is not limited to, aserver. The computing device 112 is configured to write data 116 to andread data 116 from a data store 114. The data 116 includes anyinformation that is useful for managing inventory in a single facilityor multiple facilities distributed throughout a geographic area (e.g.,city, state, country, or world). The data 116 can be provided from thirdparties (e.g., manufactures, distributors, etc.), collected on site(e.g., in a retail store at time of receipt, at time of purchasetransaction, and/or at the time of an item check using an item checkingdevice located on the sales floor), and/or derived using machinelearning algorithms.

The data 116 includes, but is not limited to, item level informationand/or display related information. The item level information includes,but is not limited to, item images, item packaging images, itemidentification codes, item locations, item descriptions, item packagingdescriptions, item regular prices, item sale prices, currency symbols,and/or sources of the items. The display related information includes,but is not limited to, display equipment images, display equipmentdescriptions, information specifying relationships between displayequipment and items, information specifying patterns of displayequipment visual states, information specifying relationships betweendisplay equipment visual states and times of day/year, informationspecifying relationships between display equipment states andalarm/notification issuances, and/or information specifyingrelationships between display equipment states and alarm/notificationverifications/challenges.

Inventory system 100 further includes POS devices 126. POS devices arewell known in the art, and therefore will not be described herein. Anyknown or to be known POS device can be used herein without limitation.The POS devices 126 can include fixed POS devices and/or mobile POSdevices (e.g., mobile phones running POS software). The POS devices 126are configured to read the markers 120 and/or labels 108 coupled toitems 118, and facilitate transactions (e.g., purchase, rent, or loan)for the items. In this regard, the POS devices capture or acquire itemidentification codes and/or transaction information during use. Thiscaptured/acquired information is used as feedback information for amachine learning function of the computing device 112 and/or imagecapture devices 104. For example, the feedback information is used torelate item identification codes with images of items captured by imagecapture devices 104 and/or information specifying item characteristics(e.g., color, size, pattern, thermal signature, etc.).

Although a single computing device 112 is shown in FIG. 1, the presentsolution is not limited in this regard. It is contemplated that morethan one computing device can be implemented. Also, the present solutionis not limited to the exemplary inventory system architecture describedin relation to FIGS. 1-4.

Referring now to FIG. 5, there is provided a detailed block diagram ofan exemplary architecture for an electronic device 500. Image capturedevices 104 and/or computing device 112 of FIG. 1 is the same as orsubstantially similar to electronic device 500. As such, the followingdiscussion of electronic device 500 is sufficient for understandingimage capture devices 104 and computing device 112.

Electronic device 500 may include more or less components than thoseshown in FIG. 5. However, the components shown are sufficient todisclose an illustrative embodiment implementing the present solution.The hardware architecture of FIG. 5 represents one embodiment of arepresentative electronic device configured to facilitate improvedinventory management. As such, the electronic device 500 of FIG. 5implements at least a portion of a method for providing machine learninginventory management in accordance with the present solution.

Some or all the components of the electronic device 500 can beimplemented as hardware, software and/or a combination of hardware andsoftware. The hardware includes, but is not limited to, one or moreelectronic circuits. The electronic circuits can include, but are notlimited to, passive components (e.g., resistors and capacitors) and/oractive components (e.g., amplifiers and/or microprocessors). The passiveand/or active components can be adapted to, arranged to and/orprogrammed to perform one or more of the methodologies, procedures, orfunctions described herein.

As shown in FIG. 5, the electronic device 500 comprises a user interface502, a Central Processing Unit (“CPU”) 506, a system bus 510, a memory512 connected to and accessible by other portions of electronic device500 through system bus 510, and hardware entities 514 connected tosystem bus 510. The user interface can include input devices (e.g., akeypad 550 and/or a camera 558) and output devices (e.g., a speaker 552,a display 554, and/or Light Emitting Diodes (“LEDs”) 556), whichfacilitate user-software interactions for controlling operations of theelectronic device 500.

At least some of the hardware entities 514 perform actions involvingaccess to and use of memory 512, which can be a RAM, a disk driverand/or a Compact Disc Read Only Memory (“CD-ROM”). Hardware entities 514can include a disk drive unit 516 comprising a computer-readable storagemedium 518 on which is stored one or more sets of instructions 520(e.g., software code) configured to implement one or more of themethodologies, procedures, or functions described herein. Theinstructions 520 can also reside, completely or at least partially,within the memory 512 and/or within the CPU 506 during execution thereofby the electronic device 500. The memory 512 and the CPU 506 also canconstitute machine-readable media. The term “machine-readable media”, asused here, refers to a single medium or multiple media (e.g., acentralized or distributed database, and/or associated caches andservers) that store the one or more sets of instructions 520. The term“machine-readable media”, as used here, also refers to any medium thatis capable of storing, encoding or carrying a set of instructions 322for execution by the electronic device 500 and that cause the electronicdevice 500 to perform any one or more of the methodologies of thepresent disclosure.

In some scenarios, the hardware entities 514 include an electroniccircuit (e.g., a processor) programmed for facilitating machine learninginventory management. In this regard, it should be understood that theelectronic circuit can access and run an inventory managementapplication 524 and/or a machine learning application 526 installed onthe electronic device 500.

The machine learning application 526 implements Artificial Intelligence(“AI”) that provides the electronic device 500 with the ability toautomatically learn and improve data analytics from experience withoutbeing explicitly programmed. The machine learning application employsone or more machine learning algorithms that learn various informationfrom accessed data (e.g., via pattern recognition and predictionmaking). Machine learning algorithms are well known in the art, andtherefore will not be described herein in detail. Any known or to beknown machine learning algorithm can be used herein without limitation.For example, in some scenarios, the machine learning application 526employs a supervised learning algorithm, an unsupervised learningalgorithm, and/or a semi-supervised algorithm. The learning algorithm(s)is(are) used to model inventory decisions based on data analysis (e.g.,captured images, item identifiers (e.g., UPCs), POS transactioninformation, display equipment information, and other information).

For example, the learning algorithm(s) is(are) configured to: detect atemporal pattern of an inventory level for a given piece of displayequipment from which predictions can be made as to when items need to bestocked and/or when floor displays need to be attended to for optimizingsales; and/or detect a temporal pattern of overall inventory managementfor a given facility from which recommendations can be made forimproving inventory management, employee management and/or storeprofitability. The machine learning algorithm(s) is(are) also configuredto facilitate the detection of item misplacements, potential theft,changes in item characteristics, changes in item packaging, changes initem popularity, and/or patterns thereof. The present solution is notlimited to the particulars of this example.

The software applications 524, 526 are generally operative to: obtainitem level information and/or other information (e.g., from markers(e.g., markers 120 of FIG. 1) and/or labels (e.g., labels 108 of FIG. 1)coupled to items (e.g., items 118)); use information from POSs, itemchecking device, and/or image capture devices located in proximity tothe POS devices to relate item identifiers with images and/orcharacteristics of inventory; compare POS transaction information withcontent of images to detect possible theft or misplacement of an item;generate an output (e.g., an alert or notification) indicating that apossible theft or misplacement of an item has been detected at a POSdevice; use the POS information for machine learning purposes; obtaindisplay related information; obtain images captured by at least camera558; process the images; store the images in a datastore (e.g., memory512 of FIG. 5 or datastore 114 of FIG. 1) so as to be associated withthe respective item level information and display related information;analyzing the images to automatically learn and update packaging changesassociated with a given item identifier; monitor display equipment(e.g., display equipment 122 of FIG. 1) in the FOV of the camera 558;track changes in the amount of items and/or types of items disposed onthe monitored display equipment; use machine learning to learn andunderstand different states and/or conditions of the display equipment(e.g., a normal condition/state, a cleanup condition/state, a restockingcondition/state, and/or a possible theft condition/state); use machinelearning (e.g., on colors, patterns, size/shape and location) to learnand understand different states and/or conditions of inventory (e.g.,properly located, misplaced, original packaging, new packaging,undamaged and/or damaged); generate alerts and/or notifications whencertain conditions/states of display equipment and/or inventory aredetected (e.g., the alert/notification comprising an indication of thequantity of an item needed for restocking the display equipment or thelocation of a misplaced item (e.g., a refrigerated and/or frozen item ona shelf outside of the refrigeration equipment)); prioritize the alertsand/or notifications based on certain criteria (e.g., total sales and/orgeographic area); cause alerts and/or notifications to be output; and/orreset data when display equipment is moved to a particular location(e.g., an exclusion area). Other functions of the software applications524, 526 will become apparent as the discussion progresses.

Referring now to FIG. 6, there is provided a flow diagram of anillustrative method 600 for machine learning inventory management. Asshown in FIG. 6A, method 600 begins with 602 and continues with 604where item level information is obtained by a computing device (e.g.,computing device 112 of FIG. 1). The item level information includes,but is not limited to, item images, item packaging images, itemidentification codes, item locations, item descriptions, item packagingdescriptions, item regular prices, item sale prices, currency symbols,and/or sources of the items. At least some of the item level informationis obtained from third parties (e.g., manufactures, distributors, etc.),collected on site (e.g., in a retail store at the time of receipt or attime of a purchase transaction), derived using image analysis, and/orderived using machine learning algorithms.

If the item identification codes are not related to the other item levelinformation and/or the relationships between the same needs to beverified, the method 600 can continue with optional 606 where machinelearning operations are performed. In 606, the item identification codes(e.g., UPCs) are associated with item images and/or other informationspecifying item characteristics based on information from POS devices(e.g., POS device(s) 126 of FIG. 1 and/or POS device 126 ₁ of FIG. 3)and/or first image capture devices (e.g., camera 104 ₇ of FIG. 3)located in proximity to the POS devices. For example, information from aPOS device includes an item unique code read from a marker (e.g., marker120 of FIG. 1 and/or label 108 of FIG. 1) and/or an item price.Information from the first image capture device includes an image of theitem that was read by the POS device. The image is stored so as to beassociated with the item unique code and/or item price. The image mayalso be analyzed to determine the characteristics of the item (e.g.,size, color, pattern, etc.). The item characteristics are then used(along with known physical characteristics of items) to relate the itemidentification code to other item related information (e.g., itempackaging images, item descriptions, item packaging descriptions, itemregular prices, item sale prices, currency symbols, and/or sources ofthe items). In this regard, the machine learning algorithm learns itemidentification codes, item pricing, item popularity, itemcharacteristics and/or item packaging characteristics associated withitems having particular item identification codes based on POSinformation. The present solution is not limited to the particulars ofthis example.

In 608, POS transaction information is compared with content of imagescaptured by the first image capture devices to detect possible theft,item misplacement, and/or learn patterns thereof. For example, an itemidentification code is used to access stored data (e.g., date 116 ofFIG. 1) specifying characteristics of the respective item and/or itempackaging. The image(s) is(are) then analyzed to detect the objectrepresented therein and determine the object's characteristics. Theresults of the image analysis are then compared to the accessed storeddata to determine if a match exists therebetween (e.g., by a certainpercentage). If a match does not exist, then a possible theft or itemmisplacement is detected. The present solution is not limited to theparticulars of this example.

If such a detection is made, then an individual (e.g., individual 130 ofFIG. 1) is notified of a possible theft or misplacement of the item. Thenotification can take any form, such as an alert (e.g., an auditory,visual or tactile alert) or message (e.g., a text message or an emailmessage). The notification can be communicated from the computing deviceto a mobile device (e.g., mobile phone 128 of FIG. 1) carried by theindividual via a network (e.g., network 110 of FIG. 1). In response tothe notification, the individual can take measures to verify thepossible theft or item misplacement, and/or dispatch the appropriatepersonal (e.g., security, police, or other employee) to address thepossible theft or item misplacement.

Next in 612, display related information is obtained. The displayrelated information includes, but is not limited to, display equipmentimages, display equipment descriptions, information specifyingrelationships between display equipment and items, informationspecifying patterns of display equipment visual states, informationspecifying relationships between display equipment visual states andtimes of day, information specifying relationships between displayequipment states and alarm issuances, and/or information specifyingrelationships between display equipment states and alarmverifications/challenges. At least some of the display relatedinformation is obtained from third parties (e.g., manufactures,distributors, etc.), collected on site (e.g., in a retail store at thetime of receipt and/or during display equipment monitoring), derivedusing image analysis, and/or derived using machine learning algorithms.

Thereafter in 614, images of at least one piece of display equipment(e.g., display equipment 122 of FIG. 1) are captured by second imagecapture devices (e.g., image capture devices 104 ₁, . . . , 104 ₆ ofFIG. 2) over a period of time (e.g., minutes, hours, days, weeks,months, years, etc.). The images are stored in a datastore (e.g., memory512 of FIG. 5 and/or datastore 114 of FIG. 1) so as to be associatedwith the respective item level information and display relatedinformation, as shown by 616. In some scenarios, unique identifiers ofthe second image capture devices are stored (e.g., in datastore 114 ofFIG. 1) so as to be associated with the respective item levelinformation and display related information at given times. The uniqueidentifiers of the second image capture devices are used to respectivelyrelate images captured thereby to the item level information and displayrelated information in the datastore. The present solution is notlimited in this regard.

The images captured by the first and/or second image capture devices areanalyzed in 618 to learn changes in item packaging. Read itemidentification codes, POS transaction information, and/or displayrelated information is(are) also used here. If such a change is detected(e.g., in size, color, shape, pattern, etc.), then the correspondingitem related information is updated to reflect the detected changes initem packaging, as shown by 620.

The second ICDs are then used in 622 to monitor the piece of displayequipment and/or the inventory (e.g., item 118 of FIG. 1) disposed onthe piece of display equipment. Images captured by the second ICDs areused in 624 to learn and track changes in the type of item(s) and/or theamount of each type of item disposed on the piece of display equipment,as well as the equipment's state of cleanliness and/or individual'sinteractions with the equipment/inventory.

The changes in item types can be identified by: identifying objectsrepresented in the images; determining the colors, shapes, patterns,heat signatures, and/or other characteristics of the identified objects;determining (if possible) an item identification code (e.g., if abarcode is visible); comparing the determined item identification code,colors, shapes, patterns, heat signatures and/or other itemcharacteristics to that contained in the pre-stored item relatedinformation; selecting an item type based on results of the comparing;and comparing the selected item type to that specified in the displayrelated information.

The changes in an item amount can be determined by: identifying objectsrepresented in the images; counting the number of visible items N_(VI)in the images (e.g., per shelf); and optionally adding to the numberN_(VI) an offset value V_(Offest) estimating the number of items whichare not visible in the images (e.g., per shelf) (i.e.,N_(VI)+V_(Offest)). Upon completing 624, method 600 continues with 626of FIG. 6B.

The equipment's state of cleanliness can be tracked by analyzing aseries of images to: detect when an item's packaging has been damaged;detect when the contents of an item have spilled; detect when an itemhas been misplaced on the piece of display equipment or in an areaadjacent to the piece of display equipment (e.g., on the ground in anaisle); and/or detect when an individual is cleaning the piece ofdisplay equipment. Misplacement of an item can be determined using thethermal signature of an item relative to the thermal signatures of theother items disposed on the piece of display equipment. In this way, theimage analysis can determine when a frozen/refrigerated item has beenplaced on a piece of display equipment absent of a refrigerator orfreezer.

Individuals' interactions with the equipment/inventory can be tracked byanalyzing a series of images to: detect when an individual takes an itemoff the display equipment; detect if and/or when the item is returned tothe display equipment; determine if the item is placed in somethingother than the display equipment (e.g., a shopping cart, a bag, a purse,a coat pocket, etc.); and/or determine if a marker/label is beingremoved from the item and/or replaced with another marker/label. Theimage analysis can also be performed to detect: when an individual isattempting to break into a locked piece of display equipment; and/orwhen an individual is attempting to break a coupler coupling an item tothe piece of display equipment. The image analysis can also be performedto detect when the piece of display equipment is being restocked by anindividual.

As shown in FIG. 6B, 626 involves using a machine learning algorithm tolearn different states and/or conditions of the piece of displayequipment and/or inventory based on the tracked changes detected in 624.The machine learning algorithm can include, but is not limited to, asupervised learning algorithm, an unsupervised learning algorithm,and/or a semi-supervised algorithm. Each of the listed machine learningalgorithms are well known in the art, and therefore will not bedescribed herein. Any known or to be known machine learning algorithmcan be used herein without limitation. The display equipment statesand/or conditions include, but are not limited to, fully stockedstate/condition, partially stocked state/condition, emptystate/condition, restocking condition/state, clean state/condition,dirty/cleanup state/condition, normal state/condition, and/or possibletheft condition/state. The inventory states and/or conditions include,but are not limited to, properly located, misplaced, original packaging,new packaging, undamaged and/or damaged.

In next 628, alerts and/or notifications are generated when certainstates and/or conditions of the inventory and/or piece of displayequipment are detected. For example, an alert and/or notification isgenerated when an item has been misplaced, an item's packaging has beendamaged, a spill has occurred in an aisle, the display equipment needsto be cleaned or restocked, the items on the display equipment need tobe reorganized, and/or a possible theft is occurring. The presentsolution is not limited to the particulars of this example.

The alerts and/or notifications are prioritized in 630 in accordancewith a pre-defined priority scheme. For example, an alert/notificationfor possible theft is assigned a higher priority than analert/notification for item misplacement, damage packaging, itemrestocking and/or item reorganization. A spill alert/notification isassigned a higher priority than an alert/notification for possibletheft, item misplacement, damage packaging, item restocking and/or itemreorganization. An alert/notification for item misplacement is assigneda higher priority than an alert/notification for damage packaging, itemrestocking and/or item reorganization. An alert/notification for itemrestocking is assigned a higher priority than an alert/notification fordamage packaging and/or item reorganization. An alert/notification isassigned a higher priority than an alert/notification for itemreorganization. The present solution is not limited in this regard.

After completing the alert/notification prioritization, thealerts/notifications are output in accordance with their assignedpriorities, as shown by 632. The output of the alerts and/ornotifications can take any known or to be known form. In this regard,the alert/notification outputs can include, but are not limited to,auditory outputs, visual outputs, and/or tactile outputs. The auditoryoutputs can include, but are not limited to, sounds output from speakers(e.g., beeps, sirens, etc.). The visual outputs can include, but are notlimited to, emitted colored light (flashing or constant), messagespresented on electronic displays, and/or information presented in aGraphical User Interface (“GUI”) of a computing device (e.g., a map, agraph, a chart, a flashing icon, etc.). The tactile outputs can include,but are not limited to, vibrations of an electronic device (e.g., amobile phone) and/or increased/decreased temperature of an electronicdevice.

As shown by 634, the images captured in 614 can also be analyzed todetect when the piece of display equipment has been moved to a differentlocation in the facility. In response to such a detection, 636 isperformed to reset the display related information for the piece ofdisplay equipment. Subsequently, 636 is performed where method 600 endsor other processing is performed.

Notably, the present solution is not limited to the particular order of604-636 shown in FIGS. 6A-6B. Operations of 604-636 can occur in anyorder selected in accordance with a particular application.

Referring now to FIG. 7, there is provided a flow diagram of anillustrative method 700 for machine learning inventory management.Method 700 begins with 702 and continues with 704 where a plurality ofimage capture devices capture images. The image capture devices are ofdifferent types. For example, the image capture devices comprise atleast one visual camera, at least one 3D camera, and at least onethermal camera. Next in 706, operations are performed to read itemidentification codes for items represented in the images. At least afirst portion of the images and known physical appearances of aplurality of items are used by at least one machine learning algorithmto learn relationships between the items represented in the images andthe item identification codes, as shown by 708. 708 may also involveusing at least a second portion of the images which were captured at aPOS to identify items represented therein based on the learnedrelationships between the items represented in the first portion ofimages and the item identification codes.

Upon completing 708, method 700 continues with 710-716. 710-716 involve:using POS transaction information to learn patterns of changes inamounts of inventory for given items over time; using the images, theknown physical appearances, the item identification codes, and POStransaction information to detect theft and learn patterns of theft overtime; and/or using the images, the item identification codes, POStransaction information, and/or stored information for display equipmentinventory to learn changes in item packaging. 710-716 also involve usingat least a second portion of the images to learn: (A) at least one of achange in a type of item disposed on a piece of equipment and a changein an amount of an item disposed on a piece of equipment; (B) conditionsof equipment or areas adjacent to the equipment; (C) individuals'interactions with equipment and inventory disposed on the equipment; (D)when equipment has been moved to a given location; and/or (E) when anitem is misplaced. In 716, patterns of (A)-(E) can also be learned. Ifthe equipment has been moved to a given location, then 718 can beperformed to reset learned information therefore. Subsequently, 720 isperformed where method 700 ends or other processing is performed.

The patterns of changes in the amounts of inventory for given items canbe detected by: analyzing POS transaction information to keep a count ofhow many units of a given item have been sold, rented, and/or loaned;and the time of year at which the transactions occurred. For example, ifa first time is sold, then a counter for the first item's inventory isdecremented, and/or a counter for the number of first item sales isincremented. At certain times of year (e.g., the holiday season), aretail store may experience increased sales of certain products. It isbeneficial to keep track of this information to optimize sales byensuring that a particular amount of inventory is at the retail storeduring those times of year.

The patterns of theft can be detected by (for example): tracking whichitems are being stolen at what times of year; and identify peak times ofyear when certain items are being stolen. This identification can thenbe used to initiate added measures to prevent theft of the certain itemsat those times of year (e.g., such as placing the product displays inareas where surveillance is increased or in areas where they will bemore exposed).

As noted above, the present solution is applicable to single facilityscenarios and multiple facility scenarios. For example, in somescenarios, retail locations in a chain usually have the same appearanceand layout. If there is a brand new Store B, the knowledge and learningfrom a similar Store A could be used to initially configure newequipment in Store B. As Store B acquires knowledge, it could fine-tuneor overlay the learning inherited from Store A until Store B's localknowledge database iteratively becomes more specific to Store B'senvironment. So not only does an inventory system learn, but it can help“train” other inventory systems which are newly installed.

All of the apparatus, methods, and algorithms disclosed and claimedherein can be made and executed without undue experimentation in lightof the present disclosure. While the invention has been described interms of preferred embodiments, it will be apparent to those havingordinary skill in the art that variations may be applied to theapparatus, methods and sequence of steps of the method without departingfrom the concept, spirit and scope of the invention. More specifically,it will be apparent that certain components may be added to, combinedwith, or substituted for the components described herein while the sameor similar results would be achieved. All such similar substitutes andmodifications apparent to those having ordinary skill in the art aredeemed to be within the spirit, scope and concept of the invention asdefined.

The features and functions disclosed above, as well as alternatives, maybe combined into many other different systems or applications. Variouspresently unforeseen or unanticipated alternatives, modifications,variations or improvements may be made by those skilled in the art, eachof which is also intended to be encompassed by the disclosedembodiments.

We claim:
 1. A method for machine learning inventory management,comprising: capturing images by a plurality of image capture devices ofdifferent types; reading item identification codes for items representedin the images; and using at least a first portion of the images andknown physical appearances of a plurality of items by a machine learningalgorithm to learn relationships between the items represented in theimages and the item identification codes.
 2. The method according toclaim 1, wherein the plurality of image capture devices comprise atleast one of a visual camera, a 3D camera, and a thermal camera.
 3. Themethod according to claim 1, further comprising using Point of Sale(“POS”) transaction information to learn patterns of changes in amountsof inventory over time.
 4. The method according to claim 1, furthercomprising using at least a second portion of the images which werecaptured at a Point of Sale (“POS”) to identify items representedtherein based on the learned relationships between the items representedin the first portion of images and the item identification codes.
 5. Themethod according to claim 1, further comprising using the images, theknown physical appearances, the item identification codes, and Point ofSale (“POS”) transaction information to detect theft and learn patternsof theft over time.
 6. The method according to claim 1, furthercomprising analyzing the images to learn changes in item packaging. 7.The method according to claim 6, wherein the changes in image packagingare further learned based on at least one of the item identificationcodes, Point of Sale (“POS”) transaction information, and storedinformation for display equipment inventory.
 8. The method according toclaim 1, further comprising using at least a second portion of theimages to learn at least one of a change in a type of item disposed on apiece of equipment and a change in an amount of an item disposed on apiece of equipment.
 9. The method according to claim 1, furthercomprising using at least a second portion of the images to learnconditions of equipment or areas adjacent to the equipment.
 10. Themethod according to claim 1, further comprising using at least a secondportion of the images to learn individuals' interactions with equipmentand inventory disposed on the equipment.
 11. The method according toclaim 1, further comprising using at least a second portion of theimages to learn when equipment has been moved to a given location, andresetting learned information for the equipment.
 12. The methodaccording to claim 1, further comprising using at least a second portionof the images to learn when an item has been misplaced.
 13. A system,comprising: a processor; and a non-transitory computer-readable storagemedium comprising programming instructions that are configured to causethe processor to implement a method for machine learning inventorymanagement, wherein the programming instructions comprise instructionsto: obtain images captured by a plurality of image capture devices ofdifferent types; obtain item identification codes read for itemsrepresented in the images; and use at least a first portion of theimages and known physical appearances of a plurality of items by amachine learning algorithm to learn relationships between the itemsrepresented in the images and the item identification codes.
 14. Thesystem according to claim 12, wherein the plurality of image capturedevices comprise a visual camera, a 3D camera, and a thermal camera. 15.The system according to claim 12, wherein the programming instructionsfurther comprise instructions to use Point of Sale (“POS”) transactioninformation to learn patterns of changes in amounts of inventory forgiven items over time.
 16. The system according to claim 12, wherein theprogramming instructions further comprise instructions to use at least asecond portion of the images which were captured at a Point of Sale(“POS”) to identify items represented therein based on the learnedrelationships between the items represented in the first portion ofimages and the item identification codes.
 17. The system according toclaim 12, wherein the programming instructions further compriseinstructions to use the images, the known physical appearances, the itemidentification codes, and Point of Sale (“POS”) transaction informationto detect theft and learn patterns of theft over time.
 18. The systemaccording to claim 12, wherein the programming instructions furthercomprise instructions to analyze the images to learn changes in itempackaging.
 19. The system according to claim 18, wherein the changes inimage packaging are further learned based on at least one of the itemidentification codes, Point of Sale (“POS”) transaction information, andstored information for display equipment inventory.
 20. The systemaccording to claim 12, wherein the programming instructions furthercomprise instructions to use at least a second portion of the images tolearn at least one of a change in a type of item disposed on a piece ofequipment and a change in an amount of an item disposed on a piece ofequipment.
 21. The system according to claim 12, wherein the programminginstructions further comprise instructions to use at least a secondportion of the images to learn conditions of equipment or areas adjacentto the equipment.
 22. The system according to claim 12, wherein theprogramming instructions further comprise instructions to use at least asecond portion of the images to learn individuals' interactions withequipment and inventory disposed on the equipment.
 23. The systemaccording to claim 12, wherein the programming instructions furthercomprise instructions to use at least a second portion of the images tolearn when equipment has been moved to a given location, and resettinglearned information for the equipment.
 24. The system according to claim12, wherein the programming instructions further comprise instructionsto use at least a second portion of the images to learn when an item hasbeen misplaced.